Patterns in Education: Linking Theory to Practice

Analysis of patterns in time and configuration (APT&C) is a measurement and analysis paradigm that bridges qualitative and quantitative research methods. APT allows analysis of temporal events (both joint and sequential occurrences), whereas APC permits investigation of structural relations or configurations. Temporal patterns can be quantified in APT, resulting in probability estimates of their occurrence. Structural relations in APC are characterized by properties that are measured quantitatively, such as: passive dependence, independence, hierarchical order, complexity, strongness, vulnerability, and wholeness. APT&C is based on theories from mathematics and general systems theory. Three empirical studies which have used APT&C are described in this article: 1) academic learning time of mildly handicapped students in elementary school learning environments; 2) patterns of mode errors in humancomputer interaction with graphical interfaces in modern software; and 3) structural configurations for supporting student autonomy in a Montessori classroom. The value of APT&C is that results can be directly related to practice. Through APT&C we have new ways of conducting educational research in order to shed light on practices that result in the outcomes we seek. The Dilemma: Qualitative vs. Quantitative Methodologies Research methods in education used for much of the 20 century were largely quantitative methods. Experimental and quasi-experimental designs were commonplace (e.g., Campbell & Stanley, 1966), and analytical techniques included ANOVA, regression analysis and their extensions (i.e., discriminate, factor, canonical and path analysis). The basic problem is that this general linear models approach seldom yielded findings that could be directly linked to educational practice. Within-group and within-person variance was often large, typically obfuscating differences between groups that could be attributed to so-called treatments, practices or programs (Medley, 1977; 1979). Cronbach & Snow (1977) further extended ANOVA to deal with aptitude-treatment interactions (ATI), with hopes of reducing the within-group variance. But this, too, was seldom successful in yielding significant results. In the 1970s and 80s, others began to explore alternative approaches that later became known as qualitative and case study methodology (cf. Guba & Lincoln, 1985; Stake, 1995; Yin, 2003). Qualitative methods have become widely used in educational research in the past two decades. One clear advantage of qualitative methods is that rich details of individual cases can give readers helpful insight into and understanding of the educational phenomena investigated. The unavoidable dilemma that often accompanies this approach is lack of justification for generalizability of findings. When samples are purposive and small, generalizability in the sense of making inferences from sample to population is seriously compromised. Indeed, respected books on qualitative methods avoid the term ‘generalizability’ and instead employ the notion of ‘transfer’ – i.e., results of what was found in this particular investigation may transfer to other similar situations the reader encounters (cf. Merriam, 1997). Mixed methods approaches have become more popular in recent years (Creswell, 2003), in which both strengths of qualitative and quantitative approaches have been utilized. Well before this, an approach that quantified qualitative patterns had been proposed: APT. Measuring System Dynamics: APT Frick (1990) proposed an analytic-measurement procedure called Analysis of Patterns in Time (APT). This is a paradigm shift in thinking for quantitative methodologists steeped in the linear models tradition and the measurement theory it depends on (cf. Kuhn, 1962). The fundamental difference is that the linear models approach relates independent measures through a mathematical function and treats deviation as error variance. On the other hand, APT measures a relation directly by counting occurrences of when a temporal pattern is true or false in observational data. Linear models relate the measures; APT measures the relation. Academic Learning Time Study Frick (1990) conducted a study of 25 systems in central and southern Indiana in which mildly handicapped children were observed throughout the day in their elementary school classroom learning environments. Each child was observed between 8 and 10 hours across multiple days over a semester. These environments ranged from self-contained classrooms for special education students to regular classrooms in which the mildly handicapped children were mainstreamed (now called inclusion). Trained classroom observers coded the kinds of academic learning activities provided, and within each academic activity the behaviors of target students and instruction made available to the student were coded at one-minute intervals. During data analysis, student behaviors at each time sampling point were collapsed into two categories: engagement and non-engagement. Similarly, instructional behaviors at each sampling point were collapsed into two categories: direct instruction and non-direct instruction. Linear models approach. As can be seen in Figure 1, if the data are analyzed with the linear models approach, student engagement can be predicted by a regression equation. Approximately 33 percent of the variance in student engagement can be accounted for by the amount of direct instruction provided. While this finding shows that there is a statistically significant positive relationship (p < 0.05) that is moderate in size, there is still a great deal of uncertainty (67 percent of the variance is unaccounted for). Notice that the vertical lines (blue) indicate the distances from the data points and the regression line (red), indicating errors in prediction. The relationship between direct instruction and engagement is represented by a function for a line. In this example, the function for the line is: EN = 0.57 + 0.40DI. Each data point represents the overall proportion of engagement for a particular student, paired with the overall proportion of direct instruction provided to that student. Engagement is aggregated separately from direct instruction for each case, so there is one overall engagement score for a student and one overall direct instruction score. Thus, there are 25 data pairs from which the regression equation is estimated. Figure 1. Linear models approach to measuring a relation APT analysis. The same data were analyzed from an APT perspective. From this perspective data are aggregated differently. The joint occurrences of student engagement and instruction were counted in order to form probabilities or proportions. For example, for student 1, the probability of (DI & EN) = 0.46; p(DI & NE) = 0.04; p(EN | DI) = 0.92; and p(EN | ND) = 0.67. These joint and conditional probability estimates for this student were based on nearly 500 data points where the joint occurrences of instruction and engagement were observed and coded. Similar probabilities were estimated for the remaining 24 systems, and then the probabilities were averaged. Thus, there were nearly 15,000 data points representing the joint occurrences of direct instruction and engagement in the 25 systems. See Table 1. Table 1. Temporal Relationships: Joint Occurrences of Direct Instruction (DI), Student Engagement (EN), Non-direct Instruction (ND), and Student Non-engagement (NE) in Columns 3 6; Conditional Occurrences in Columns 7 8. p(DI) p(EN) p(DI & EN) p(DI & NE) p(ND & EN) p(ND & NE) p(EN|DI) p(EN|ND) 0.50 0.80 0.46 0.04 0.34 0.16 0.92 0.67 0.39 0.49 0.37 0.02 0.12 0.49 0.95 0.20 0.27 0.56 0.26 0.01 0.30 0.43 0.97 0.41 0.34 0.69 0.34 0.00 0.35 0.31 1.00 0.53 0.48 0.73 0.47 0.01 0.25 0.26 0.98 0.49 0.40 0.75 0.39 0.01 0.35 0.25 0.98 0.59 0.44 0.84 0.40 0.04 0.44 0.11 0.91 0.80 0.36 0.75 0.33 0.03 0.42 0.22 0.92 0.65 0.30 0.67 0.29 0.01 0.39 0.32 0.96 0.55 0.32 0.71 0.31 0.01 0.40 0.29 0.98 0.56 0.42 0.68 0.42 0.00 0.26 0.31 0.99 0.46 0.38 0.84 0.37 0.01 0.47 0.15 0.97 0.75 0.31 0.63 0.31 0.00 0.32 0.37 1.00 0.46 0.54 0.87 0.52 0.02 0.36 0.11 0.97 0.77 0.81 0.92 0.81 0.00 0.11 0.08 1.00 0.57 0.67 0.77 0.62 0.05 0.15 0.18 0.93 0.45 0.24 0.76 0.24 0.00 0.52 0.24 1.00 0.69 0.34 0.74 0.34 0.00 0.40 0.25 0.99 0.61 0.59 0.87 0.58 0.01 0.29 0.12 0.99 0.71 0.52 0.64 0.48 0.04 0.16 0.33 0.93 0.33 0.62 0.83 0.58 0.04 0.25 0.13 0.94 0.66 0.23 0.65 0.22 0.01 0.43 0.34 0.97 0.56 0.29 0.79 0.28 0.01 0.51 0.20 0.97 0.71 0.54 0.75 0.52 0.02 0.23 0.24 0.97 0.49 0.51 0.82 0.50 0.00 0.31 0.18 0.99 0.63 Mean (SD) Mean (SD) Mean (SD) Mean (SD) Mean (SD) Mean (SD) Mean (SD) Mean (SD) 0.432 (0.144) 0.741 (0.101) 0.416 (0.139) 0.015 (0.010) 0.324 (0.114) 0.243 (0.104) 0.967 (0.029) 0.573 (0.142) When direct instruction was occurring, these students were engaged on average about 0.967 (SD = 0.029). In the absence of direct instruction, their average engagement was about 0.573 (SD = 0.142). In other words, such students were 13 times more likely to be off-task during non-direct instruction [ (1 – 0.573)/(1 – 0.967) = 0.427/0.033 = 12.94 ]. APT measures the temporal relation between direct instruction and student engagement. The error in prediction is indicated by the standard deviation of the mean percent of time. For the p(EN|DI) mean (0.967), the standard deviation was quite small (0.029). In a normal distribution, 95 percent of the scores will fall between 1.96 standard deviations above the mean and 1.96 standard deviations below. For the p(EN|ND) mean (0.573), the standard deviation was much larger (0.142), indicating much more variation in student engagement when non-direct instruction was occurring. In other words, student engagement was not only lower but it was less predictable when non-direct instruction occurred, compared with occurrences of direct instruction. When analyzing the same observational data in APT framework, the conclusions are very clear. For these mildly handicapped students, if direct instruction was occurring, then they were very likely to be on-task, and this was very consistent across all students, regardless of the specific learning environment in which they were observed. During non-direct instruction, students were much more likely to be off-task, though this was less predictable. These results have direct implications for practice. More details are provided in Frick (1990). It should be noted that observational data were collected temporally. In this case, joint occurrences of student and instructor behaviors were coded at one-minute intervals (time sampling). Alternatively, sequences of behavior or events can be observed and coded, as illustrated in the next study, reported below. If observations are not made in these ways, it is not possible to conduct APT. As can be seen in Table 1, when independent measures are obtained as would normally be the case under traditional views of measurement, one would only have the data in the first two columns for analysis. This is how measurement has been traditionally conceived: measure things separately, then relate the measures by means of some kind of function. A measure of association is used to indicate the relationship, such as a Pearson Product Moment Coefficient. Columns 2-8 in Table 1 were created because an APT perspective was taken in the design of how the relation was coded by observers (between student engagement and instruction). One would not have the remaining columns of data for analysis had these variables been measured independently. Only because an APT perspective was taken was it possible to estimate the conditional probability of the relation, p(EN|DI), for each target system. Moreover, APT measures of relation can be aggregated and subjected to standard methods of analysis, such as computation of means and standard deviations, as well as linear regression, illustrated in the academic learning time study by Frick (1990). Patterns of Mode Errors in Human-Computer Interfaces This empirical study by An (2003) illustrates the value of the merging of qualitative methods and quantification via APT. She investigated conditions of mode errors when people use modern software. Mode errors occur when the same user action results in more than one outcome, depending on the context. Mode errors can cause serious problems for software users, such as inadvertent destruction of important work, decreased productivity, and task incompletion. Sixteen college students were each asked to perform eight computer tasks during usability tests of three modern direct-manipulation software interfaces. Stimulated recall interviews were conducted immediately afterwards as subjects watched themselves on videotape to clarify why they took certain actions during the tests. An observation system was devised for coding tapes of usability tests to record behavioral patterns of the participants. Qualitative analysis of the results indicated three major types of mode errors: A) right action, wrong result; B) it isn’t there where I need it; and C) it isn’t there at all. A source of error analysis revealed that mode errors appear to result from eight categories of design incongruity: unaffordance, invisibility, misled expectation, unmet expectation, mismatched expectation, inconsistency, unmemorability, and over-automation. Consequences of mode errors included: can’t find hidden function, can’t find unavailable function, false success, stuck performance, inhibited performance, and inefficient performance. Analysis of patterns in time (APT) was used as a quantitative method to determine the likelihoods of temporal patterns of types, sources and consequences of mode errors. Results of queries regarding temporal patterns were as follows: Table 2. Sequential patterns of mode errors, their sources, and consequences. Query Relative Frequency Likelihood (p) Type A 1 IF type of mode error IS right action, wrong result, 34 out of 52 0.65 a) AND IF source of mode error IS unaffordance, 15 out of 34 0.44 THEN consequence IS can’t find hidden function OR false success? 10 out of 15 0.67 b) AND IF source of mode error IS invisibility, 6 out of 34 0.18 THEN consequence IS stuck performance? 5 out of 6 0.83 c) AND IF source of mode error IS misled expectation, 7 out of 34 0.21 THEN consequence IS false success? 6 out of 7 0.86 Type B 2 IF type of mode error IS it isn’t there where I need it, 8 out of 52 0.15 a) AND IF source of mode error IS mismatched expectation, 8 out of 8 1.00 THEN consequence IS can’t find hidden function? 8 out of 8 1.00 Type C 3 IF type of mode error IS it isn’t there at all, 10 out of 52 0.19 a) AND IF source of mode error IS unmet expectation, 10 out of 10 1.00 THEN consequence IS can’t find unavailable function? 10 out of 10 1.00 b) AND IF source of mode error IS unaffordance, 3 out of 10 0.30 THEN IF source of mode error IS unmet expectation, 3 out of 3 1.00 THEN consequence IS can’t find unavailable function? 3 out of 3 1.00 Query 1a consists of three phrases. Phrase 1 (IF type of mode error IS it isn’t there where I need it) occurred for 34 out of the 52 mode errors observed (p = 0.65). Given that Phrase 1 was true, Phrase 2 (AND IF source of mode error IS unaffordance) was observed to be true for 15 out of those 34 occasions [ p(Phrase 2 | Phrase 1) = 0.44) ]. Given that both Phrases 1 and 2 were true, Phrase 3 (THEN consequence IS can’t find hidden function OR false success?) was observed to be true for 10 out of those 15 occasions [ p(Phrase 3 | Phrase 2 | Phrase 1) = 0.67) ]. This is interpreted to mean that when users did the right action but got the wrong result, the source of user error was often software interface elements that lacked affordance (functionality that is not obvious). This frequently resulted in users being unable to find a software function that was hidden from view, or they thought they did the task correctly only to find out later they had not (false success). When these conditions for modes occurred, software users in this study were unsuccessful 67 percent of time in tasks they were trying to do. While a larger random sample of a broader range of users beyond college undergraduates would be needed to increase generalizability, these findings nonetheless illustrate measurement and analysis of sequential patterns of mode errors and their estimated likelihoods. These findings offer useful guidance to software designers who want to make their products easier to use by minimizing sequences of human errors such as those observed in this empirical study. While this particular study does not examine patterns of teaching and learning that would be of more direct interest to educators, it does illustrate the value of a difference in approach to measurement. This study demonstrated the practical value of mixed methodologies where one first uses qualitative methods for gaining understanding of patterns and relationships, and then uses APT to code and quantify those temporal relationships in a manner that is useful to practitioners for making decisions based on APT predictions. Further information on formal definitions of APT sequences and associated pattern counting rules can be found in Frick (1983), Chapters 2 and 5 (note that APT was originally called nonmetric temporal path analysis but the name was changed to prevent confusion with conventional path analysis; only the name has changed). Measuring System Structure: APT&C Thompson (2006) provided the significant insight that APT could be extended to characterize structure or configuration of educational systems, in addition to characterizing system dynamics – or processes in education – as APT was designed originally to do. Frick and Thompson have since extended APT to measure and analyze configurations in education (APT&C, 2005). Configural patterns characterize structures in education – i.e., how education is organized, or relations between parts. Axiomatic Theories of Intentional Systems (ATIS) provides the theoretical foundation for quantitative measures of system structure required by APT&C (Thompson, 2006). These measures include: complexity, hierarchical order, heterarchical order, compactness, centrality, flexibility, active dependence, passive dependence, independence, interdependence, strongness, unilateralness, weakness, wholeness, and vulnerability. ATIS is a systems theory that predicts relationships among system properties, both structural and dynamic. There are over 200 axioms and theorems in ATIS. For example, #106 predicts: If system strongness increases, then toput increases. See Thompson (2006) and ATP&C (2005) for further details. Koh (2005) was engaged in research on student autonomy. She realized that ATIS properties and APT&C would help the analysis of ethnographic classroom observational data she planned to collect. Measuring structural properties of systems is a way to characterize classroom structure that supports student autonomy. Student Autonomy Structures in a Montessori Classroom Autonomy or self-determination was defined by Deci, Vallerand, Pelletier & Ryan (1991) as a state where volition for action is totally internalized and determined by intrinsic motivation and not by external conditions. According to self-determination theory, intrinsic motivation cannot be fostered if autonomy support was lacking in social environments. Experimental studies and self-reported surveys conducted with school-age children found that perceived autonomy had a positive impact on perceived competence, intrinsic motivation and conceptual learning of school-age children (Grolnick & Ryan, 1987; Valas & Sovik, 1993; Hardre & Reeve, 2003). The Montessori system aims to educate each child towards self-mastery and independence (Montessori, 1964). A distinguishing feature of Montessori classrooms is its provision for student autonomy. In a study comparing the social context of Montessori and traditional middle schools, Rathunde & Csikszentmihalyi (2005) found that Montessori students reported more support from teachers, more order in the classroom and spent more time with academic work rather than in passive listening. Koh’s (2005) case study explored how classroom structures support student autonomy in a Montessori classroom. Ten one-hour observations were conducted in April, 2006, in an upper elementary Montessori classroom located in southern Indiana. It had twenty-eight students, ages 10-12, a Montessori-certified head teacher and two assistant teachers. Data on interactions between teachers, students and classroom resources were collected through ethnographic field-notes. The constant comparative method (Creswell, 1998) was used to identify common interaction patterns and classroom activity structures. Measures of structural configurations were determined using definitions from ATIS. For example, one definition is: An investigator constructs a directed graph representing the affect relations in the system, consisting of vertices (v) and edges (e). The degree (d) of a vertex is the number of edges that touch it, and the degree of initiating edges is the number whose direction is leading from a vertex, (dI(v)). |Ai| is the cardinality of the set of components in the affect relation, where i indexes the different types of affect relations in the system. See reports for further technical details on structure measures and ATIS (APT&C, 2005). Koh (2005) was interested in the structure of affect relations concerning ‘choice of work’ and ‘guidance of learning’. Apparent from classroom observations and confirmed through teacher interviews, there were two clearly different activity structures in the mornings. Students normally started each day with a new Head Problems worksheet created by the teacher, consisting of math and logic-related problems. When this was completed (usually within an hour), they typically spent the next three hours in the Morning Work Period, during which students chose the type of works they wanted to engage in. Works constituted the major part of their learning goals: research projects related to physical science, natural science, history and geography; and book reports, science experiments and math workbooks that students needed to complete each nine-week period. The Morning Work Period supported student autonomy with respect to which works they wanted to do, and whether they wanted to work on them individually or collaboratively. This morning activity pattern of initial Head Problems followed by the long Morning Work Period was consistent in all ten observations. The data depicted in Figure 2 are from a typical configuration selected from one morning, since this configuration was relatively stable from one day to the next. While the specific works chosen by students tended to differ each time (as well as the daily worksheets for Head Problems), the activity structures with respect to the ‘choice’ and ‘guidance’ affect relations were highly similar (homeomorphic). ‘Centrality’ measures the number of indirect connections from each primary initiating component (i.e., one with a directed edge from that component which does not have a directed edge to it) to all others. During Head Problems, ‘centrality’ was found to be substantially higher since the teacher chose the same activity for all students to work on. In comparison, during the Morning Work Period the students chose 0 10 20 30 40 50 60